How Well Do Vision Transformers (VTs) Transfer to the Non-natural Image Domain? An Empirical Study Involving Art Classification

Vision Transformers (VTs) are becoming a valuable alternative to Convolutional Neural Networks (CNNs) when it comes problems involving high-dimensional and spatially organized inputs such as images. However, their Transfer Learning (TL) properties not yet well studied, is fully known whether these neural architectures can transfer across different domains CNNs. In this paper we study VTs that pre-trained on the popular ImageNet dataset learn representations transferable non-natural image domain. To do so consider three well-studied art classification use them surrogate for studying TL potential of four VTs. Their performance extensively compared against common CNNs several experiments. Our results show exhibit strong generalization networks more powerful feature extractors than